Skip to content

test(e2e): run conformance in gateway lanes - #2925

Open
elezar wants to merge 4 commits into
mainfrom
codex/standalone-conformance-cli
Open

test(e2e): run conformance in gateway lanes#2925
elezar wants to merge 4 commits into
mainfrom
codex/standalone-conformance-cli

Conversation

@elezar

@elezar elezar commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Run the standalone OpenShell conformance suite in every gateway-backed e2e lane, using a conformance CLI artifact built once for CI and supplied alongside the prebuilt OpenShell CLI. This replaces the removed Rust smoke coverage with gateway-agnostic conformance checks for Docker, Podman, Kubernetes, and VM gateways.

Related Issue

Changes

  • Added the reusable openshell-conformance scenario and runner library.
  • Added the distributable openshell-conformance CLI with list, run, JSON output, default-all scenario execution, and optional explicit scenario selection for direct CLI use.
  • Moved the previous portable smoke coverage out of the Rust e2e crate and into the standalone conformance library.
  • Added a shared e2e helper that requires OPENSHELL_BIN and OPENSHELL_CONFORMANCE_BIN instead of building binaries implicitly from test wrappers.
  • Updated Docker, Podman, Kubernetes, and VM e2e wrappers to run conformance against the instantiated gateway before any lane-specific Rust tests.
  • Updated mise tasks so local gateway e2e tasks build the conformance CLI once through a hidden dependency, while CI keeps --skip-deps and fails loudly if the prebuilt conformance binary is missing.
  • Added CI build/download wiring for the conformance artifact alongside the existing e2e CLI artifact.
  • Documented standalone conformance CLI usage.

Testing

  • cargo test -p openshell-conformance -p openshell-conformance-cli
  • bash -n on the e2e wrapper scripts and shared conformance helper
  • mise tasks --json
  • mise run --dry-run e2e
  • mise run pre-commit
  • Confirmed successful CI rust-docker and rust-vm-managed lanes downloaded OPENSHELL_CONFORMANCE_BIN and printed the conformance run banner before executing the smoke scenario.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Documentation updated

@elezar
elezar requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners August 25, 2026 13:34
@elezar
elezar force-pushed the codex/standalone-conformance-cli branch from 15ce6f3 to 2db17a8 Compare August 25, 2026 14:03
@elezar
elezar force-pushed the codex/2183-cli-conformance-minimal branch from e794b2c to a257f6a Compare August 25, 2026 14:21
@elezar
elezar changed the base branch from codex/2183-cli-conformance-minimal to main August 25, 2026 14:22
@elezar
elezar force-pushed the codex/standalone-conformance-cli branch from 2db17a8 to 4c31c98 Compare August 25, 2026 14:31
@elezar
elezar changed the base branch from main to codex/2183-cli-conformance-minimal August 25, 2026 14:31
@elezar
elezar force-pushed the codex/standalone-conformance-cli branch from cfdf625 to 6e7c13b Compare August 26, 2026 11:48
@elezar elezar added the test:e2e Requires end-to-end coverage label Aug 26, 2026
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

Label test:e2e applied for 6e7c13b. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@elezar

elezar commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

/ok-to-test 6e7c13b

@elezar
elezar force-pushed the codex/standalone-conformance-cli branch from 6e7c13b to b6a240f Compare August 26, 2026 12:15
elezar added 4 commits August 26, 2026 14:36
@elezar
test(e2e): isolate VM-specific smoke assertions
ddedcd0 
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
test(e2e): add portable CLI conformance baseline
1ba067b 
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
feat(conformance): add standalone CLI runner
a764f85 
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
test(e2e): run conformance in gateway lanes
2a2e632 
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the codex/standalone-conformance-cli branch from b6a240f to 2a2e632 Compare August 26, 2026 12:38
@copy-pr-bot

copy-pr-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@elezar elezar mentioned this pull request Aug 26, 2026
Closed
15 tasks
@elezar elezar changed the title feat(conformance): add standalone CLI runner test(e2e): run conformance in gateway lanes Aug 26, 2026
@elezar
elezar changed the base branch from codex/2183-cli-conformance-minimal to main August 26, 2026 13:02
@elezar

elezar commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

/ok-to-test 2a2e632

@elezar
elezar enabled auto-merge August 26, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrunalp mrunalp Awaiting requested review from mrunalp mrunalp is a code owner
@sjenning sjenning Awaiting requested review from sjenning sjenning is a code owner
@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr derekwaynecarr is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@elezar